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ABSTRACT 

This booklet begins by examining the role played by 
evaluation in course and curriculum development, and then shows how 
the basic "error elimination" approach advocated by the philosopher 
Karl Popper can be used as a basis for the on*going evaluation of 
instructional systems. Next, two contrasting paradigms of evaluation 
are described, one that concentrates on measuring the outcomes of the 
instructional system (the agricultural/botanical or scientific 
approach) and one that pays more attention to what happens during the 
educational process itself (the social/anthropological or 
illuminative approach). Finally, five diagnostic techniques commonly 
used as part of an evaluation strategy are reviewed and the 
respective uses, strengths, and weaknesses of each are discussed: (1) 
results from student assessment; (2) student questionnaires and 
interviews; (3) observation of the instructional system in progress; 
(4) feedback from teaching staff directly involved with the 
instructional system; and (5) feedback from people having an indirect 
link with the instructional system. Models of the systems approach to 
course development, the role of an instructional system, and the 
general methodological approach advocated by Popper are provided, as 
well as extracts from a Likert scale and a semantic differential 
scale taken from course evaluation questionnaires. An annotated list 
of three items recommended for further reading is included. (MES) 
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The Role of Evaluation in 

Course and Curricuium Development 



Introduction 



This booklet deals with the subject of evaluation. It begins by 
examining the role played by evaluation in course and curriculum 
development^ and then shows how the basic 'error elimination* 
approsch advocated by the philosopher Karl Popper can be used as 
a basis for the on-going evaluation of instructional systems. Next, 
the booklet describes two contrasting paradigms of evaluation - one 
that concentrates on measuring the outcomes of the instructional 
system (the so-called agriculturallbotanical or scientific approach) , 
and one that pays more attention to what happens during the 
educational process itself (the so called social I anthropological or 
illuminative approach). Finally, it reviews the range of diagnostic 
techniques which are commonly used as part of an evaluation 
strategy and discusses the respective uses, strengths and 
weaknesses of each technique. 



The role of evaluation in instructional design 

In the booklet on 'Education objectives', it was shown that the 
process of course or curriculum development can be represented 
schematically by figure 1 . 

As can be seen» the process is basically cyclic in nature, with the 
first three stages being: 

(i) the formulation of a clear set of objectives for the course or 
curriculum; 

(ii) the selection of appropriate instructional methods for achieving 
these objectives within the context of the course or curriculum; 

(iii) the implementation of the course or curriculum. 

In the booklet on 'Student assessment', we started to examine the 
fourth and final stage of the process - the assessment and evaluation 
stage. In that booklet, we distinguished between assessment and 
evaluation, showing that the two terms (which are often used virtually 
synonymously in common pariance) have radically different 
meanings when used in an instnjctional context. 

We defined assessment as 'those activities that are designed to 
measure student learning achieved as a result of a teaching/learning 
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situation't and evaluation as a 'serias of activities that are designed 
to nneasure the effectiveness of a teaching/learning system as a 
whole'. However, we noted that the results of student assessment 
may well form part of the wider evaluation process. 




Rgure 1 : schematic representation of the systems approach to course or cur- 
riculum development 

Within the systematic approach to instructional design that is shown 
in figure 1, the role of on-going monitoring and evaluatio*^ of the 
system is of vital importance to its development and evolution. 
Because of the cyclical and interactive nature of the system, each 
cycle can benefit from the experiences and feedback obtained from 
previous cycles. Evaluative feedback can be gained from a wide 
range of sources and via a wide range of methods, and, in many 
cases, a whole battery of evaluation techniques are used in order to 
gain an overall view of the effectiveness of the system in question. 
Whether this is a complete course, part of a course, a particular 
teaching session, a self -instructional programme, or a teaching aid 
such as a film or video, the designer (or team of designers) should 
never be happy wuh their first attempt, or even with revised versions. 
If one takes the view that 'the system can always be iniproved', 
on-going evaluation should always be an integral part of the design 
process. 
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The scopo and depth of the evaluation that is carried out in any 
particular case will vary according to the nature of the situation, as, 
indeed, will the evaluation methods usud. Whatever the circum- 
stances, however, the importance of using appropriate evaluation 
procedures to monitor the instructional system and provide the basis 
for improvements cannot be underestimated. Feedback obtained 
from such evaluation should shed light on the appropriateness of the 
teaching methods used, the structure adopted, the implemer)tatiQr) 
strategy ,x\\e studer)t assessmer^t methods, and even the aims and 
objectives themselves. With each successive cycle of the system, 
the teaching/learning situation should become progressively more 
finely 'tuned' and should consequently become more efficient and 
more effective through a continuous process of evolution and 
improvennent. 



Instructional development by enror elimination 
- a 'Popperian' approach 

The philosopher Karl Popper originally used the concept of 'error 
elimination' to explain how progress is made in developing scientific 
theories. The same concept can be applied to the development and 
improvement of instructional systems of all kinds. 

Underlying rationale 

The 'error elimination' approach to the development of instructional 
systems is based on two assumptions: 

(a) that an instructional system is not an independent entity, justify- 
ing its existence a priori, but is part of a total system - fulfilling a 
specific function by helping to get from Situation 'A' shown in 
figure 2 to Situation 'S'. 



Situation A 



Situation B 



People with the 
potential to play 
a certain role In 
society once they 
have acquired the 
necessary skills 
and knowledge. 



Instructional system 
designed to supply all 
or part of the 
necessary education 
and/( r training. 



Qualified people 
capable of playing 
that role In society 
or progressing to 
more advanced study 
or training on the 
road to attaining 
their ultimate goal 



Figure 2: the role of an instructional system 

(b) that the development and improvement of the instructional sys- 
tem is most effectively tackled by adopting the general 



methodological approach proposed by Karl Popper, an ap- 
proach that can be summarised by the schema shown in figure 
3. 



1 

Initial problem 
situation 

(Identification of 
the need for the 
Instructional 
system) 



TS - 
trial solution 



(development and 
operation of the 
instructional 
system) 



EE - 

error 
elimination 

(looking for 
ways in 
which the 
instructional 
system is 
falling to 
achieve its 
objectives) 



new problem 
situation 

(identification 
of areas In which 
the instructional 
system could be 
improved) 



Rgure 3: the general methodological approach advocated by Kart Popper 

Stages in the development of an instructional system 

We see that there are four general stages in the above approach. 

Stage 1: The Identification of the Initial 'problem situation' (P^ ). 

This itself can be seen as having three sequential stages: 

(a) Identification of the desired objectives (knowledge, skills and at- 
titudes); let us call thece *X\ 

(b) Identification of the relevant knowledge, skills and attitudes al- 
ready possessed by the prospective students; let us call these 

(c) Identification of the objectives represented by *X-Y\ the gap to 
be bridged by the instructional system. 

Stage 2: Development and operation of the Instructional system (TS) 
This falls into two stages: 

(a) Designing an instructional system capable of achieving the ob- 
jectives represented by 'X-Y* (or at least a part of the differ- 
ence). This involves developing the overall structure, selecting 
and sequencing the content, choosing appropriate teaching 
methods, and so on. 

(b) Making appropriate administrative arrangements to put the in- 
structional system into operation (i.e. implementation of the 
system) . 

Stage 3: The 'error elimination' process (EE) 
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This stage involves carrying out a critical examination and analysis of 
Stages 1 and 2. It is the key stage in Popper's methodology, 
according to which a new instructional system can be regarded in the 
same way as a new scientific theory which has been developed in an 
attempt to resolve a specific problem situation but which has not yet 
been subjected to rigorous experimental testing. According to 
Popper, such a theory should be tested not by trying to prove it right 
(an impossible task from a logical point of view) but by trying to 
prove it wrong, i.e. by looking for specific ways in which the theory 
can be shown to be incompatible with experimental evidence. In the 
case of a new instructional system, the testing should be carried out 
not by trying to prove that is is succeeding in achieving its objectives 
(a very difficult thing to do with any degree of rigour) but by looking 
for ways in which it is manifestly not succeeding (a much easier 
task). Needless to say. such an approach requires a healthy attitude 
towards criticism that is sometimes lacking in those who develop and 
operate educational and training courses; all too often, these try to 
defend their course against criticism by contrived arguments and 
rationalization rather than accepting valid criticism and attempting to 
rectify the situation through improvements to the instructional 
system. 

Stage 4: Identification of the new * problem situation' CPg ) 

If carried out correctly. Stage 3 should reveal areas in which the 
instructional system needs to be improved, and (hopefully) point to 
how these improvements might be carried out. It therefore leads to a 
new problem situation. 9^ , that can form the starting point of a 

further development cycle. 

TS EE P3 

Thus, Popper's methodology is seen to be both open-ended and 
on-going, forming a basis for the continuous development of 
instructional systems of all types. 

How the error elimination process may be carried out 

There are two basic questions that should be asked of an 
instructional system: 

(a) Are there any ways in which the instructional system is mani- 
festly failing to achieve its design objectives? 

(b) Are there any ways in which the organisation and logistics of the 
instructional system are unsatisfactory? 



Finding answers to question (a) is essentially a long-term process 
and can be done: 

(i) by surveying students who have undergone the instructional sys- 
tem (i.e. former students), and 

(ii) by surveying people who are not directly involved in the instruc- 
tional system under scrutiny, but who nevertheless may be able 
to make relevant comments and observations (e.g. employers 
who subsequently take on the students, or the teachers and or- 
ganizers of any subsequent courses or training situations to 
which the students proceed) . 

Finding answers to question (b) is usually easier, and can be done: 

(i) by surveying the staff who are involved in implementing the in- 
structional system, and 

(ii) by surveying the students who are involved in the system. 

Techniques through which the above information can be obtained will 
be discussed later in this booklet. 



Two contrasting paradigms of evaluation 

A major area of debate in educational evaluation is concerned with 
the relative merits of two distinctly contrasting approaches. Op the 
one side, there is the so-cailed agricultural/botanical approach, 
which reflects a 'scientific' approach to evaluation; on the other, 
there is the social/anthropological approach, which is more 
concerned with the hidden processes which occur during an 
educational experience. The latter approach has become known as 
'Illuminative evaluation*. 

The agricultural/botanical approach to evaluation 

The agricultural/botanical approach has its origins in scientific 
experiments set up to assess the effects of specific variables (the 
nature of the soil, fertilisers, etc.) on the grov/th of crops. Such 
experiments have tight controls, and the resulting outcomes can 
generally be measured relatively easily. When applied to education, 
this approach has led to the use of systematic, objectives-oriented 
evaluation procedures. This 'traditional' strategy sets out to measure 
thr extent to which a given instructional system has achieved certain 
specific goals (its objectives) in relation to the students* 
pre-kncwiedge or existing skills. To this extent, the agricultural/bo- 
tanical evaluation paradigm measures output against input, and often 
treats the differences statistically. Other factors in the system, such 



as the learning environment, teaching personnel, course content and 
structure, and teaching methods normally receive only incidental 
examination, if they are considered at all. This general 'systems 
approach' has been used when measuring the relative efficiency of 
different methods in teaching towards a common end, and also to 
measure the effectiveness of self-instructional programmes in 
achieving stated objectives. 

Thtt social/anthropological approach to avaluation 

By comparison, the social I anthropological approach is more 
concerned with studying the on-going process of education. In 
general, the techniques used are far more subjective, and often 
involve personal value judgements of the results. The arguments in 
favour of this type of approach are that the variables in educational 
developments cannot be readily identified or controlled, and that 
'inputs' and 'outputs' can be varied, complex, difficult to specify with 
certainty, and often virtually impossible to measure. In such cases, 
the evaluator explores the perceptions, opinions and attitudes of staff 
and students, using a variety of methods, in an attempt to reveal 
what was otherwise hidden in the educational process. The 
evaluation process is generally not rigidly structured or constrained, 
and usually gives the evaluator scope to follow up specific areas of 
interest as and when they become apparent. Illuminative evaluation 
of this kind has often been referred to as 'attempting to open up the 
black box of the educational process'. 

Comparison of the two approaches 

These two basic paradigms of evaluation differ significantly both in 
their methodologies and in their treatment of results. Thry also differ 
in their focus. The agricultural/botanica! approach is basically 
designed to find out if specified goals have been achieved. The 
social/anthropological approach, on the other hand, is more flexible, 
and is designed to find out what has been achieved and why. 

Clearly, there must be some middle ground between what, on the 
one hand, purports to be a purely objective approach, and the 
largely subjective approach that is embodied in illuminative 
evaluation. Where the correct balance lies, however, depends to a 
large extent on what is being evaluated, and for what purpose. A 
useful reviev; of how appropriate evaluation strategies can be 
matched with different types of educational development has been 
given by Tory Becher (see review article cited in 'Further Reading' 
section at end of booklet), and interested readers are referred to 
this. 




A review of evaluation techniques 



As should be evident from the previous section, there is no single 
correct way to conduct an evaluation exercise. One may, for 
example, be looking for outcomes (whether intended or not) in 
cognitive, affective and skills areas, and also for an insight into 
possible problems concerning the implementation ano operation of 
an instructional system. Much also depends on whether one is 
adopting an 'illuminative* strategy or a more rigid 'objectives-based' 
approach. 

Because of the variety of information that one may be seeking during 
an evaluation, it is normally advisable to use an appropriate battery of 
evaluation techniques. Some of the possible information sources are 
listed below: 

(a) results from student assessment; 

(b) student questionnaires and interviews; 

(c) observation of the instructional system in progress; 

(d) feedback from teaching staff directly involved with the instruc- 
tional systen: 

(e) feedback from people having an indirect link with the instruc- 
tional system. 

Each of these sources of feedback generally has an important part to 
play, regardless of whether the evaluation is of a course or unit of 
teaching that is still in the process of development (formative 
evaluatlor)) or of a fully-developed instructional system that is 
already in use {summatlve evaluatlor)). Let us now examine each 
source in more detail. 

(a) Results from student assessment 

When an instructional system has sharply-defined objectives, a 
critical study of the results obtained from student assessment can be 
of great assistance in the error elimination process described above. 
Two basic techniques can be used. 

(i) Arialysis of studerit assessmerits that form a riormal part of the 
hstructlonal system 

When student assessments are an integral part of a course or 
other instructional system, the results of and trends indicated by 
these assessments can usually shed considerable light on the 
operation of the system as a whole. The evaluator should, as a 
result, be able to judge which objectives are being well 
achieved, and, more importantly, which objectives are nof. 
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When students do not perform as well as expected, there is a 
traditional tendency to conclude that it is oasically the fault of 
the students. This may occasionally be the case, but, more 
often than not, there are other factors involved. A systematic 
approach to instructional design allows all aspects of the system 
to be analysed, and may reveal that there are in fact a number 
of reasons for unsatisfactory student achievement, e.g.: 

• the teaching methods were not well matched to the course -ob- 
jectives; 

• there were problems in the operation of the instructional sys- 
tem; 

• the assessment methods used were not suitable: 

• the objectives themselves were not realistic. 

Critical analysis of this sort allows the instructional system to be 
continuously monitored and progressively 'tuned'. 

(11) Analysis of student assessn^Bnts carried out solely for evaluation 
purposes 

When an instructional package of some sort is being trial tested, 
or when the relative effectiveness of two methods is being 
measured, specially-designed student assessment techniques 
can be used to evaluate the effectiveness of the methods in- 
volved, rather than to assess the students themselves. Such 
cpproaches are normally essentially 'agricultural/botanical' in 
nature, and often involve the use of pre- and Dost-tests, 'con- 
trol' groups, and the statistical analysis of differences. They are 
one of the standard methods of evaluating new systems, tech- 
niques, packages, etc. 

(b) Student questionnaires and interviews 

Obtaining feedback from students regarding their experiences with 
and their opinions of an instructional system is one of the most 
common approaches to evaluation. The information can be sought 
through questionnaires and/or Inten/lews, and can be treated either 
objectively or in a more illuminr*'ve manner. Student feedback can 
be obtained through a variwty of so-called *self-reporting' 
techniques, several of which have been adopted from the field of 
attitude measurement. Let us now examine some of the more 
important of these techniques. 

LIkert scales 

Essentially, a LIkert scale is an attitude measurement instrument 
consisting of a list of statements, the person responding having to 
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make a judgement on each statement, often selecting one response 
Irom a number of degrees of agreement and disagreement. A typical 
example is shown in figure 4. 





strongly 
agree 


Agree 


Disagree 


strongly 
disagree 


1 . 1 find the course easy 










2. The course contains 
too many lectures 










3. The course does not 
Include enough 
practical work 










4 ThA Course orovldes 
satisfactory 
facilities for 
Individual tutorials 










5. 











Figure 4: part of a Likert scale used In a course evaluation questionnaire 

The number of points on such a scale depends on the specific 
requirements of the setter, although the use of an even number of 
options has the advantage of making it impossible for students to 
*duck the issues' by repeatedly taking refuge in a completely neutral 
category. 

In practice, it is harder to produce 'good' statements than it first 
appears, and some trial testing of the statements may well be 
necessary. Indeed, there is a fair amount of skill associated with 
preparing statements which are vd//c/ (see booklet on 'Student 
assessment'), and which, at the same time, provide good 
discrimination. 

Likert scales can be used to monitor students' general opinions of an 
instructional system. It is also possible to use such statements for 
comparative purposes, e.g. by pre- and post-testing the students, 
or by comparing an 'experimental' group with a matched 'control' 
group. 
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Semantic differential scales 

A sen^antlc differential scale is a somewhat different type of attitude 
scale, consisting of v;ord pairs of antonyms joined by a 3, 4, 5, 6 or 7 
point scale. The method is b:3ed upon the premiss that 'he word 
pairs are opposites. although this may not always be valid in 
practice, because of the fact that particular words sometimes have 
different meanings for different students. Part of a typical semantic 
differential scale of the type used in course evaluation is shown in 
figure 5. 



/ consider the course to be: (mark appropriate box on each row of scale) 



easy 














difficult 


inflexible 














flexible 


too 

theoretical 














too applied 


poorly 
structured 














well 
structured 



















Figure 5: part of a semantic differential scale used \n a course evaluation 
questionnaire. 

Objectives rating 

Student ratings of the degree of achievement of learning objectives 
is sometimes used in student feedback questionnaires. Here, the 
objectives of an instructional system are listed, and the student is 
asked to indicate whether each objective has been 'well achieved' 
through to 'not achieved at air. The rating is generally carried out 
using a five-point scale, but variations are possible. This type of 
scale is particularly useful in cases where no other suitable technique 
exists for measuring the achievement of certain objectives, or as a 
cross-check on other evaluation techniques. 

Free student comments 

If students are allowed to respond freely on topics raised in a 
questionnaire, unexpected outcomes and attitudes may often 
emerge. Although it may be difficult to categorise free responses, 
these should normally be sought as a matter of principle, since they 
can often add a completely new dimension to an evaluation. 



Interviews with students 

Student interviews are basically a verbal fornn of student 
questionnaire. A well-run interview can, however, probe nnore 
deeply and sensitively into specific areas of interest than can 
normally be done in a written questionnaire. One drawback is that 
Individual interviewing is a tinne-consuming procedure. Thus, the 
most effective role of sampled interviews may well be to check the 
validity of a more widely-used formal questionnaire. 

(c) Observation of the instructional systems in progress 

An understanding of the hidden educational processes occurring 
within an instructional system may sometimes be developed by 
means of careful and sensitive observation of these processes. The 
observation can be direct and immediate, or may be recorded in 
some way (e.g. on videotape) for later analysis. Such techniques 
are particularly useful when one is evaluating exercises designed to 
develop communication and interpersonal skills. The ethical 
problems associated with 'unobstrusive assessment' that were 
mentioned in the booklet on 'Student assessment' are not really a 
problem in this case, as it is the instructional system which is under 
scrutiny, not the students. 

(d) Feedback from teaching staff directly involved 
with the instructionai system 

Through questionnaires, interviews and solicited commonts, the 
opinions of staff directly involved in the implementation and operation 
of an instructional system can be of great value in course evaluation. 
Their comments may be influential in evaluating aU aspects of the 
system, including the validity of the objectives, the cjurse structure, 
the teaching sequence, the assessment methods, and the 
day-to-day organisation and management. 

(e) Feedback from people having an Indirect link 
with the instructional system 

People who do not have a direct link with the actual teaching/learning 
system under investigation may still be able to make an important 
contribution to its evaluation. Again, questionnaires, interviews and 
solicited comments are appropriate means of gathering information. 

The advice of employers, for example, may be sought if a vocational 
course is being evaluated. This may be done at the formative 
evaluation stage, before a course has been fully devoloped (in order 
to assess the skills and qualities which employers are looking for 
from students). It may also be done as part of a summative 
evaluation process (to gather information on the relevance of the 
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course to the actua! work situation and on the general strengths and 
weaknesses of former students) . 

Similarly, the comments of former students can be important, since 
they are in a position to comment on the relevance of the course or 
other instructional system in retrospect, and perhaps suggest 
improvements with the benefit of hindsight and experience. 

If a course has external examiners, their comments are invariably 
extremely influential in course development. Such feedback will 
probably prove even more valuable if the external examiners 
concerned are given some guidance as to which particular aspects 
should be commented upon, however. 

Finally, the opinions of teachers who subsequently take on a 
particular group of students for a related course are often highly 
relevant. Their comments on the students' strengths and 
weaknesses may be important when revising a particular instructional 
system, or part thereof. 



Conclusion 

Evaluation is a valuable, and, indeed, essential component of the 
process by which the on-going development of instructional systems 
occurs. The evaluator has a wide range of techniques at his disposal, 
and also has several relevant sources of feedback which may assist 
in compiling a 'total picture' of the system and its effects (both good 
and bad). No single approach is the best in all circumstances, and it 
is generally most profitable to use a battery of appropriate evaluation 
techniques to ensure that the overall evaluation process is as valid 
and useful as possible. 



Further Reading 

f . Evaluation and educational technology, by T. Becher. In Aspects 
of Educational Technology XV, edited by F. Percival and H. I. 
Ellington; Kogan Page, London; 1981. (An extremely useful re- 
view of the general field of evaluation.) 

2. Measuring and Evaluation in Teaching, by N. E. Gronlund; Mac- 
millan. New York; 1971 (one of the definitive texts on the sub- 
ject ) 

3. The Evaluation of Instruction, by N. C. Wittrock and D. E. Wiley; 
Holt, New York; 1970 (Another definitive text on the subject.) 
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